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effectiveness. The procedure employed was to review the literature on 
the subject of library evaluation. Each criterion or measure of 
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in terms of its potential significance and validity for measuring 
library performance. The first section of this report contains a 
discussion of some of the factors Involved in measuring library 
performance and an outline of existing evaluation methods. The second 
section is devoted to discussing individual studies and the 
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which it is believed additional research will produce sound method (s) 
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EFFECTIVENESS CRITERIA AND MEDICAL LIBRARIES 
1 . 0 INTRODUCTION 

Today, as always, libraries of all types are confronted with a 
number of problems involving an assessment of their performance. Some of 
the problems relate to the evaluation of services or functions for which 
there are no reliable methods of measuring the quality of service or function. 
One fundamental problem is that none of the current evaluation methods seem 
to consider total library performance as Gritioal to making a valid evaluation 
Another proKLem is that most of the present evaluation methods are not 
completely acceptable to either librarians or non-librarians because none 
of the methods seem to be sufficiently sensitive to both quanitative and 
qualitative factors of library service. 

This is a report on a study undertaken for the National Library of 
Medicine*; the study objectives were to develop a list of issues and criteria 
that relate to the problem of measurement of medical library effectiveness. 

To the extent it was deemed appropriate other types of libraries wqre 
considered as there exists a rather broad common bond between all types of 
ilbraries. The procedure employed was to review the literature on the subjeot 
of library evaluation. Each criterion or measure of evaluation that was 
encountered a list of criteria and was examined in terms of 

* The study was conducted from August 1969 to April 1970 under a grant 
(IGOI^ Lfl 007818- 01) from National Library of Medicine and the 

:5outhwGSt Reg^^^ (Uhlversity of ealifornla, Los Angeles, 

^^.Blomedical-Xihrary),, vvV V 



its potential signi.f ieanee and validity for measuring library performance. 

Hie puipQse of the listing and examination was to aid in the process of 
defining areas of needed research and to recommend areas where work could be 
conducted with soma expectation of good results and high returns. In 
addition, . the study conducted an analysia of a raeently completed medicri 
school library evaluation project carried out for the National Library of 
Medicine by the University City Science Center. This work involved some 
additional statistical analysis of the UCSC data. 

In the first section of this report^ we diacuaa some of the factors 
involved in measuring library performance and outline existing evaluation 
methods. The second section is devoted to discussing Individual studies 
and the evaluation criteria that were used. In the third section we discuss 
areas in which we believe additional research will produce sound method(s) 
of evaluating totri library performance. ^fhis third section is without 
doubt hi^ly subjective and open to question; however, if these suggestions 
generate some debate and some work, they will have served a useful purpose. 

The final section is devoted to the analysis data from the University 

City Science Center’s Final Report - National Survey of Medical School Libraries 

1-1 Factors in the Evaluation of Medical Libraries 

In order to acoomplish the objaGtlves set forth in the proposal to the 
National Library of Medicine, the authors examined over five hundred articles, 
books and abstr acta relating to the evaluation of libraries. Perhaps the 
most aurprielng aspect of the literature review was ah apparent lack of 
concern with the how or the why of the evaluation process. It would seem to 
be self-evident that any discussion of, library perform should be prefaced 



by a discussion about what aspects are to be evaluated, how they were 
selected and why they are to be evaluated. There are a surprising number 
of reports and studies on the subjsot of evaluation that fail to make it 
clear just what the intended purpose was, and as a result, confusion arises 
over the interpretation of the resiilts. 

Another rather surprising finding was the lack of concern about the 
total service program of a library. Most of the studies, with presumably 
sound reasons (al though seldom spelled out) confine themselves to one or 
two evaluation criterion as applied to one or two service functions. While 
no single study can cover all services, some attention should. In our 
opinion, be given to the matter of how the services studied or evaluated 
fit into the total service program of a given library or type of library. 

In general this total service concept was lacking, and so it was difficult 
to determine whether the criterion was appropriately selected and employed. 

It was also notable that those few studies that did concern themselves 
with the full range of services all failed to consider one rather basic 
function of all libraries. In the very broadest sense the library's main 
function is that of dissemination; however, in terms of usual considerations 
the library has two functions: immediate dissemination and conservation for 

later dissemination. None of the studies examined concerned themselves with 
the question of aonBervatlon. VJhlle it may be true that only the large 
teaching-research- regional medical library may. n®6fl tb; be significantly 
concerned with oohservatiun, all libraries must hate some concern about the 
matter. In general all Me material acquired by the library, even the small 

housedyin the library because there is some expectation 



that the material mil be used and used by a number of different people over 
a period of time. If this were not so, the material would simply be given 
to the person requiring the information in much the same manner that a good 
deal of the inter-library loan materials (photocopies) are now handled. 

The point being made is that many methods of evaluation place a high premium 
on the performance of a service or function that is or may be detrimental 
to the conservation of the materlals--an equally Important library function. 

Perhaps the clearest example of the above problem is in terms of 
reader accessibility to materials. Most libraries assume open stacks are 
important in allowing the greatest possible use of the materials. Open 
stacks store books in an environment that Is most comfortable for people 
(a 65-70° temperature with 50-60% humidity and ample natural or artificial 
li^t) and most detrimental to the documents. In a closed stack situation 
the materials could be stored in conditions that would slow down the chemical 
and bacterial action that destroys most of the present day papers. It is 
also possible that such a move would not change the' performance of a library, 
especially if the performanGe were evaluated in terms of response time. An 
example of this would be the Document Delivery Test (DDT) of the UCSC study. 
In this test, a premium is placed on response time and any document not on the 
shelf at the time the test is given lowers the library’ s performance score. 

In a closed stack situation fewer items woiild be misshelved or in use with 
no record of who i So using it or where it is being used. Items in the 
categories "missing” Vor ”ean*t locate" are given a long response time score 
in the Document Delivery Test, thus in this case, a high DDT score is 
positively correlated with conservatipri. There are a humber of queations 




about the usefulness of such a test and some of these are discussed in a 
later section (4.0). 

One of the most comprehensive projects on library evaluation was 
i conducted for the Army Technical Libraries by the John I. Thompson Company 
during Ibub, 1967, and 1968.* This project conducted an extensive literature 
search up to 1967. The ATLIS investigators came to the conclusion that 
most studies had focused on one or two factors and ignored the implications 
for the total system. Our work carrying the literature search up to 
January 1970 indicated that the situation has not changed. However, even 
the ATLIS studies, though they are rather comprehensive, do not consider 
the conservation aspect of the problem. Admittedly the special-technical 
library is the least likely to have to consider this problem. We will 
return to this matter later in the section on needed research. 

Our literature search revealed that most studies of library performance 
coneentrated on one or two service functions. Only a very few studies 
attempted to review the entire scope of library services and those that did 
ignored one or two areas of concern. (It also indicated that no study 
examined the question of conservation as an issue, or factor, in library 
performance . ) 

2.0 STUDIES OF LIBRARY PERFORMANCE 

Shortly after starting the literature reviet^ it became apparent that 
ij^jwe tried to set up a separate category for each slightly different method 

need one category for each study examined. In 

* Weasel, C. J. Criteria for Evaluating the Effectiveness of Library 

. Operations and ^Services ^ Atlis reports 10, 19, 21. Was : 

John I. Thompson and Company, 1967, 1968, 1969. 



order to provide the reviewers with a framework to organize their work a 
list of about sixteen measures was developed. After the review was finished, 
it became rather clear from the material covered that there was still a high 
degree of overlap. Because of the overlap we decided to further reduce 
the list by considering only the methods of analysis that were employed. 

By this means, we drew up a list of six "basic" criterion of library 
performance or effectiveness. The list given below represents the order 
in which we will present the discussion of the individual studies. 



Basic Criterion for Evaluation Library Performance 

I. Accessibility* 



II. Cost 

III. Use 

IV. User Satisfaction 

V. Response Time 

VI. Cost-Benefit 



In the ensuing pages we will discuss some of the studies that have 
either used or suggested using one or more of the above measures as a method 
of evaluating library performance. A research stucJy is conaldered under 
as many topics as necessary . At; the end of each unit there will be a 
discussloh of the measures listed in that section and some indication of 
tlje'ir research potential. The discussions are not intended to summarize the 
results ‘of the reported study or its methodology. Our only purpose is 



to analyze the probl^^ potential utility of the various measures that 






* , See section 3.0 for discussion of specific measures included under each 
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have been suggested 



The use o£ citations to specific studies is 



to document the sources for measures discussed. 

Evaluation studies of library services are generally concerned with 
reader services rather than technical services. Studies of technical 
services seem to be considered matters of internal concern. Nevertheless, 
there have been a number of studies conducted on various aspects of technical 
service work. In many ways technical service studies point up a basic 
v^eakness noted for all evaluation studies j that is, they are not concerned 
with the question of what will happen to reader services as a result of 
changes in technical services (i.e. , they are not concerned with the, total 
library operation) . Studies evaluating services may be considered as 
being one of two types; a) how efficient the organizational pattern is 
in terms of the various operations required to carry out the library 
objectives, and b) how efficient or effective the staff is in providing 
the services or performing the required tasks to meet the library 



studies seem to be concerned with determining what the goals or objectives 

each one is or the degree to which a specific aervice 



above generalizatlbri ; however, even In that instance there is no 




It should be noted that almost without exception none of the 




attempt to. determine the relative weight each service has in terms of 
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2*1 Acnessibllity 



Acceeaibillty to the library and its contents ia without a doubt one 
of the most difficult criteria to measure* Of the criterion discussed^ 
accessibility factors are the least often considered. There are two 
aspects to the gueation of access • (a) physical access to the library 

and its materials , that is the ease i^lth which one may determine if a 
particular document is in a ccllection and where it is located^ and 
(b) to what class of user is a given service available. To a large extent 
studies of the organization and form of the public catalog are accessi^ 
bility studies 5 although they are seldom considered in this light. In 
order to measure physical accessibility in a quantitative manner^ it 
becomes necessary to consider response time^ but as will be noted most 
studies that discuss this problem fail to take this variable into account. 
At the same time most studies that deal with problems of response time 
fail to consider the problem of who has access to the service. A total 
approach to even one segment of the problem of evaluation of library 
performance seems to be the exception rather than the rule. 

The problem of location and availability of materials has been 
V 0 xplpred^^b Bush and Worse (1956) by gathering data on cases of fa_Llure 
to meet demands for material held by the library, A ratio of materials 
used to materials demanded was the criterion employed to determine 
efiactrvChessv^ The" informs tio on failureT.to meet demands was strongly 
influenced by of the materials and the length of the loan 

period. In essence the study was concerned x-jith accessibility of the 
materials collection . -There i^as also some considercation of the time 
delay such was one of .a notable 
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exception in that they did consider the problems of response time. As 
with all the criteria discussed there is an overlap, in this case there 
is a very clear element of user satisfaction that should be considered. 

Again, this points out the need for a total approach to the problem of 
performance evaluation. 

In discussing accessibility, Fusaler (1961) calls frequency of use 
the major factor in producing increased efficiency in the organization 
and housing of materials. Fussier (1951) considers accessibility essential 
and suggests a better understanding of readers’ needs and working methods 
Is the only manner by which a library will be able to improve user access. 

In other studies the criterion for evaluation is the ratio of the number 
of books to the size of the user population. This method was proposed for 
use by the U. S. Office of Education in choosing academic libraries to 
receive grants in aid. Clapp (1965) argues that minimum adequacy of academic 
libraries can be measured by the number of volumes, and, considering such 
variables as student body size and composition, faculty size, cirriculum 
and campus location, he develops formulas for estimating the number of 
books required. In an article on college libraries, Weber (1957) suggests 
that ease of access and the number of successes achieved is more important 
for evaluation purposes than the volume count. The underlying assumption 
in studies of this type is that each user should have access to a minimum 
number of /documents. Wf'ber points o employing this 

concept is that the number of successes is more relevant for evaluation 
than mere volume counts. Of equal importance or perhaps even greater is 
the question of quality and relevancy. Access to e. million volumes will 



ERIC 






.■• 9 - 






1 







be of no use to a* patron 1£ the million volumes are not of interest to 
that person, even if he can receive any one of the milllan documents in 
a fraction of a second. 

Another factor of importance to the above discussion is the class of 
user who is given access to the documents or services being evaluated. 

Very few studies have made any attempt to explore this problem area. One 
study that concerned itself with this, and a wide range of other services, 
and the patron groups that received these services, was the Orr, et.al. 
(1968c) study. This is one of the few studies that relates specifically 
to the medical library situation. Their measure of effectiveness 5.s called 
the "Standardized Inventory of Library Services''. An attempt is made to 
create a scale that will allow the library to assess its services in terms 
of what the official policy is and to provide a means of comparing the 
results betvjeen institutions. The method suggested for scoring the 
inventory is based upon having a number of people allocate points on the 
basis of personal opinion as to the value of a given service. User 
satisfaction would seem to be the criterion of evaluation that is being 
suggested in this study. Yet what the study reveals is the degree to which 
various user classes are given service. Until such time as the full text 
of the ILS is published, one must withhold any true evaluation of their 
technique (see Section 4.5 for a discussion of the ILS as repor ted in the 
UCSG Final Report') . 

■ . • ■ .■■■ - ...... • . ' 

seems very clear that there is need to consider the question of 



accessibility in the process of evaluating any library. To ignore the 



studies have done is to bury one's head in the sand. 
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Rapid response times, lov^ costs, high use rates, good cost-benefit ratios, 
and great user satisfaction may not be a reflection of the ideal library 
performance goal. A situation such as Just described could probably be 
achieved today in more than 99% of the existing libraries without an 
Increase in their budgets. Obviously there is a missing element in the 
above formulation and that is the matter of accessibility and who and what 
is to be serviced. Libraries could achieve rapid response time, for 
example, at the expense of cutting back on services to "marginal" users 
and for "marginal" materials. There has aj.ways been the problem of 
determining potential and margir.,al users and most libraries do very little 

^ r 

in this area. Therefore any system of performance evaluation that ignores 
the question of accessibility, or at least availability, will only compound 
the problem by encouraging program administration to concentrate upon 
the achievement of the optimum performance for the known factors, and thus 
in fact sub-optimize the total system performance. 

2,2 Costs 

The re has always been a concern vjith the problems of cost control 
in the library. Yet throughout most of the history of library development, 
librarians have approached the problem of cost analysis in a most elementary 
and timid: mariher . Much of the reluctance can no doubt be attributed to 
a lack of training in cost analysis on the librarian’ s pert and a lack of 
concern over the level and slgnlficarice of dollars being expended on the 
p^aft, of, library control boards and the general public. It is only recently. 




demonstrated on anyone’s part about unit costs and cost control. The 



question of cost control and library performance evaluation are clearly 
connected. Fortunately, moat cost factors are relatively easy to identify 
and quantify. 

The literature contains, a great many general cost studies and most of 
them cannot be employed directly in evaluating the performance of the 
library. The studies discussed are merely samples of the material that 
is available. In general the smaller and more standardized the analyzed 
unit Isi the more suitable it is for use In evaluating the performance of 
the library through time or for comparing the performance of a particular 
task between libraries. 



In order to determine more accurately the basic unit costs of 



technical services, Brutoher (1964) recommends using cost accounting 
techniques. Library functions are divided into cost centers for such 
services as ordering, cataloging, book selection, etc., with derivative 



figures showing the cost of having a book on the shelf is equal to the 



average purchase price plus the average cost per book processed. Risk 
(1956) outlines the means of applying cost accounting methods to the 
library and measuring the total cost to determine what the out- 

put should be in terms of activities and personnel and using the standard 



work hour measure. While these cost figures cannot be used to evaluate a 

■ ■ ■ ■ ■ ' ■ -i,- ‘ ■ ■ 

library, they are useful in determining the efficiency of some operations 
and when used in combination with Other criterion can begin to provide 

■ ■■■- ■ ■ -■ i;- • ■■■■ - . 

some insight into library performance. 

Bouman (1951) points out that the value of a company . library , or 

■ ' ' - ' ' ' _ > \ 

any other library, should not be based on production costs alone, but 



rather on the replacement value. To establish the cost of internal 



operations and provide a bfasis for future budgeting the Technical Services 
Division of the University of Denver Library was studied by Wynar (1963) . 
In his summary of the results, time and cost factors on a per book basis 
are given for the order department, cataloging department, and mechanical 
preparations, with the total cost of preparing a new non-fiction title 



averaging $4. 33, Relating book funds to users, the Knox (1959) study found 



that an average of $30 per professional user (in this case chemists) vjas 
spent each year on materials. Niland (1967) attempts to develop a method 
for evaluating budget proposals of libraries,. Using multiple regression 
methods he attempted to set up some standards of evaluating proposed 
budgets and develop realistic requests. Again none of these approaches 
will completely solve the problem of evaluation | however, each one does 
have some merit and in conjunction \-jith each other can provide a reasonably 
clear picture of the cost factor in library operations. 

Another aspect of the cost picture that is often Ignored is the 
personnel requirements, especially when dealing with public services. 

There seems to be a great reticence and reluctance to put a "price tag" 



or unit cpst" figure on public service activities . Admittedly the ''value" 
of the service is difficult to measure, and the unit cpst figure may be 
high which may create some problem in "demonstrating" the desirability 
of such a service. However, of her variablea, such as the quality of a 



coilection, are studied and occasionally even from a cost point of view. 
In- any case, any method of evaluation or standards of service that specify 
or suggest certain skills, abilities, training, etc., carry with them some 









basic cost factors,. that should, not be 









Many studies have focused on the number of staff required and the 
qualifications, special training or skills needed to perform a service 
adequately. It is generally recognized that an arbitrary number of staff 
members cannot be set but must depend on the kind and scope of service 
provided, the content of the collection, and the size of the organization 
served. The ALA College Library Standards prescribe a minimum number 
of profeasional librarians with additional hiring determined by size of 
population served, type of library organization, size and character of 
the collection, prevailing community interests, number of hours the library 
is open and arrangement of the building. In 1964 the Special Libraries 
Association suggested a ratio of 2-3 non-professionals to every professional 
library staff member. In a criticism of the SLA standards, Randall (1965) 









recommends a ratio of one staff member for every 100 potential or 75 
actual library users. In considering the qualifications of library staff, 
there is agreement on such requirements as graduation from a library 
school for professionals and sue!, generalities as loyalty, modesty, and 
the ability to tvork v^ith people. Brophy (1960) summarizea two personality 
studies and discusses the attributes of a model librarian. He suggests 
the creation of work situations to fit individual abilities. Others have 
defined staff qualifications as the skills necessary for performing out- 
lined responsobilities. The duties of a reference librarian are listed 



in Hu, f ford (1950) who suggests . tha’t continued schooling and active membef- 



ship in professional organizations as a means of improving skills. This 

- , s , . - ^ .• 

.y: /V/'" ■ '■' ' ‘ ' '■ '■ ■ ’ ■■ ■ ' ' '■ ‘ ' 

approach-^ requires giving time off from regular duties and thus Increases 



(1950), presehts definite qualifications in what he calls a 
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realistic approach to special librarianship. These include expert 



knowledge of the sources of information and reference tools, training in 
administration, a year course in acquisitions and cataloging, a subject 
specialization and working knowledge of French and German. Other studies 
have discussed the qualifications of the non-professional staff and . 
suggested in-library training through the use of staff manuals. As 
can be seen very little hard research has been conducted in the area 
of the number and kind of personnel needed to perform a library service, 
nor in the area of the type of skills and knovjledge that lead to better 
performance. Decisions are made on the basis of educated guesses. 

In general the approach to evaluating performance in terms of cost 
carries with it the assumption that the lowest cost figure represents 
the ideal provided all other things are equal. To be perfectly blunt, 
the rather primitive state at which the investigation of library services 
novj finds itself in does not provide a feeling that any investigator 
can show that "all other things" are in fact equal. Cost analysis and 
unit costs are valuable management tools and can, if used alone and 
with available data, he used to evaluate a small aegment of a given 




procesa. The writers believe that the more effective way of using the 
cost criterion for evaluating overall library performance is as part of 
a comprehensive weighted system employing all the evaluation criterion 




in the form of a thorough cost-benefit analysis or its equivalent . 




.some time. The level of sophistication in the nature of the units of 
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! use considered up until recently has been very lm\?. Normally the unit 

employed is the number of documents circulated. Occasionally the number 
of registered borrowers has been used. As the number of circulations 
or borrowers increase, the level of performance is assumed to also raise. 
Recently a number of new approaches have been employed that represent a 
great improvement in evaluating the validity of the use criterion. 

I One approach to measuring library performance is through examining 

the use it receives. A library with materials and services designed to 

? meet the users’ needs (there are some that are primarily archlval- 

I museum collections) is not effective unless it is used. Use studies 

have been conducted in a number of areas and for a number of purposes. 

In evaluating utility, per capita circulation la often mentioned as an 

p; 

I indicator of the effectiveness of the library system. A study of 

j circulation patterns at Arnold Engineering Development Center (Randall 

I 

J 1SS6) showed a ratio of one actual user to four potential users and a 

i ’ ■ ■ ■ ■ ■ 

\ direct correlation between job complexity and library use. An elaborate 



five-year survey of a technical library (Strain 1965) serving 4,000 
people yielded a ratio of one actual user to two potential users, as 
well as the months of high and lot? circulation points, circulation- 
s collection ratios and the average number of books on loan at any one 

time, alliindlcators of utility. Although intended for other purposes 
Steig's study (1942) and Trueswell's (1965) work have been Used by some 
people to evaluate overall performance. Both studies are concerned with 
the degree and frequency of book collection use . 
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;; Looking to the future, ^3orse's C1&64, 1368) study of the MIT Science 

f Library attempts to predict through circulation figures, the amount of use 

I the library will receive. The major work (196S) is a very detailed study 

I relating to library effectiveness and use. An operations research 

I orientation is employed throughout the book and so many librarians will 

have difficulty in applying the very useful evaluatlon-deeislon-making 
L models that are supplied. 

In a criticism of the 1964 SLA standards, Randall (1965) proposes 
i that the ratio of circulation to acquisition should be six to one, and 








that the circulation figures should be equal to 90% of the total collection 
and there should be a yearly circulation rate of 20-30 items per user. 

In evaluating with circulation figures, it should be remembered that 
in-house use of the materials is not usually included. A good summary of 
work in the methodology that may be employed in conducting use studies 
can be found in the Lehigh University (1965) report. The Middle States 
Association of Colleges and Secondary Schools (1958) in an article on 
what; criteria should be considered in evaluating libraries says that the 
use of the library is the ultimate test of its effectiveness. 

Use of the Individual collections has also been employed to determine 
the relevance of the collection to a user needs. This in turn is considered 
as a measure of the library performance capability. Evaluation of materials 
on the basis of use has been applied to the problem of weeding by Trueswell 






i 



i 



I 



(1964, 1965, 1966, 1969) . Using quantitative., methods, Trueswell has 



B V considered the last circulation date as an indication of circulation 



requirements and shown that holdings might be reduced by 60 to 70% and 
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still satisfy over 99% of the users’ requirements. The assumption is a 
smaller active collection would be easier to use and coat less to 
maintain (better performance) . 

In a description of a special library’s weeding policy, Bedsole (1958) 
lists circulation data as one of the criteria to be used. In this study 
he describes a policy of weeding out any book not borrowed in the Last 
seven years. Several others have employed the last data used approach; 
Fussier (1961) , Lister (1967), Trueswell (1965), and Jain (1965). In the 
Bush (1956) study, the measure of effectiveness chosen was the ratio of 
material used to material demanded. Material used was material borrov;ed 



plus material consulted in the reading room and material demanded was 
material used plus material unavailable. The library i^as found to be 
39% effective by this measure. Two factors must be considered when 
evaluating this approach: a) a large number of requests are the result 

of a very fev; users, and b) at best this reflects only successful search- 
use patterns and does not reflect users who failed in their search. 

Again, as we pointed out earlier, it Is relatively easy to reduce cost 



'or increase use, but in so doing, will the library still be able to provide 
the full range of library services? ' -v , 






Another approach to partially evaluating the collection and the 



rlibf on thiB: basis of use ;is . described, by 






Cole (1958) . This method 









enfiploys an analysis. of the- reference questions asked " including both those 
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performance from these data. There V 70 uld always be an element of doubt 

in that only active user needs would be apparent and even these 

users may only ask some of their information questions of the library. 



f There are several other studies relating to use that should be noted. 

I Of these the Meier (1961) study is most interesting. Unfortunately, the 

I published article fails to provide enough data to make a completely 

I: 

j adequate evaluation of his method. It seems likely that a good deal of 

i the basic input for the University City Science Center’s Document Delivery 

I . 

I Test was drawn from this source. In his study, Meier approaches the 

I problem of use from a more total library evaluation point of view than do 

I 

I most other studies encountered. In addition, Meier recommends inter- 

library comparisons with the use of his item-use-day unit. His explanation 

of this unit of measure shoves that it is operational, has minimal ambiguities, 

I can be agreed upon by similar institutions and can therefore be used for 

; comparison. Another unit of measure proposed by Ohlman (1965), although 

: t ' ' ' ' ' '' ' 

I not suggested for comparisons, is the activity ratio obtained by dividing 

‘ the number of potential transactions completed during one month by the 

number of potential transactions available at the end of that month. 



Schutze (1952) also suggests using circulation figures and user classification 
by type as a basis for new book selection or duplication. 

There are a ; number of problems that arise from the employment of the 

use criterion to measure library performance: 




it fails to differentiate between types of use 
non-signif icant) 



(signif icant ;:and 




it does not include ’’in house’’ use , for such data canno.t be 
collected without employing expensive data collection methods.. 




c) it la susceptible to radical variations as a result of a 
change in use patterns of a small percentage of the total 

user population If this group is In the very active user category. 

d) it fails to reflect any information about the potential users 
who have either not attempted to use the library or who have 
given up as a result of repeated failures to satisfy information 
needs. 



With all its drawbacks, the use criterion is the only criterion that 
begins to provide a Gompletely objective measure of the total performance 
situation. Costs are objective, but they fail to relate the relevance of 
the performance to even a portion of the user population. Cost-benefit 
analysis appears to be rigerous and objective, but is frequently forced 
to use subjective benefit factors such as a user satisfaction criterion. 
Response time is also an objective measure but like cost analysis, it 
fails to provide the evaluator vjith information on relevance. Thus 
use, v/hile not terribly satisfactory, can be considered at least a partial 
index of relevance and of a library’s ability to provide needed services 
to some portion of total potential user population. 



2.4 User Satisfaction 

As indicated above, measuring user satisfaction is a highly personal 
process. Because of the subjective nature of such a criterion, one must 
be very careful about drawing eonclusions about the performance of one 
or several libraries that are too sweeping in scope. Within the broad 
heading of user satiafactipn, there are two distinct sub-groups to be 
considered: (a) user satlsfactlori \i?ith existing services and materials, 

and (b) attempts to determine user needs for services and materials not 
presently available. 
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Libraries have always attempted to evaluate their document collections 
in some manner; we have already noted the efforts to determine unit costs 
for materials and the studies that evaluate the collection in terms of 
use. Another method that has always had wide currency has been in terms 
of comparing a particular library’s holdings to some "ideal" value or 
list. We decided to place this type of study under the broader category 
of user satisfaction rather than accessibility primarily because of the 
manner in which the results of such studies are moat frequently used. 

That is the results are usually made known to the patrons and they are often 
asked to note items on the list that were not held but which should be 
purchased. It is possible, although it seldom happens, that some of the 
items that were not held are not desired. In this way the users, or at 
least some of the users, are able to indicate their satisfaction or lack 
of satisfaction with the library’s holdings. 

A survey article by Hirsch (1959) describes four methods of evaluating 



a collection: 1) the impressionistic method usually done by scholars, 

2) the checklist method, 3) on the basis of circulation and use data, 

4) comparison of expenditures for materials with expenditures of other 
institutions. A common technique is the comparison of library holdings 
with a list of "Ideal" holdings in a certain field. Examples of this 
approach are Emerson (1957) and Downs (1942 , 1965 , 1966) ; although these 



studies were undeftaken for different purposes, the method was the same. 



The evaluation of -ithe collection was made in terms of the percentage of 




items held .when a,j bibliography is checked against; the catalog. The 
publication source, language distribution and time span of the citations 




may alao be analyzed and the library holdings are checked against the 
list as a measure of adequacy. Similarly, Maizell (I960) recommends as 
a measure of the quality of the technical library collection, a comparison 
with published lists of key literature or citations in the moat used 
journals, as well as a record of unfilled requests received by the 
library. 

Impressionistic evaluation is seen in Anderson’s (1965) study of a 
multi-level file structure where subject specialists examined a random 
sample of documents drawn from a larger number to determine the reliability 
of the file, i.e., the collection. Many studies have been done on the 
relevance of the retrieved documents to the user of an information system; 
for example, Cleverdon (1958, 1959, 1962, 1964), Bornstein (1061), 

Cuadra (1964), The value of these studies lies in testing the efficiency 
of the indexing system, although the same methods can be used in the 
evaluation of materials. However, several fundamental questions remain 
unansvjered. Foremost among these is what is relevant, and does that 
definition adequately serve all users? We, too, leave these questions 
unanswered. 

Determining user requirements is most important as an aid to 
selection, weeding and evaluation of materials. Scientists* attitudes 
and methods of gathering information were examined through questionnaires 
and surveys by Allen (1966) , Atherton (1962) , Bare (1966) , Bourne (1961) , 
Lehigh University (1965) , and Tomlinson (1965) . All studies in this group 
iwere oriented towards basing a user needs. 

Bach (195?^) also analyzed" the information needs of scientists by noting 



the number of journals required, the age and language of journals. He 
also differentiated between pure and applied scientists. In these studies 
the question to be answered is how well are the known needs being met? 

Classifying users by type is another approach toward identification 
of the potential collection requirements and for meeting user needs. 

Bernal (1960) desGrlbes this method, considering user function as the 
determining factor; he distinguishes between the researcher, the practical 
user, the user writing articles, the science historian and the chance 
reader, each type requiring a different kind of material. Slater (1963) 
does the same for an industrial library, noting the research methods 
used. General criteria for judging the value of a piece of Information 
in technical library materials selection were delineated by Mayo-Wells 



(1964) . These Included currency, authenticity, reliability, assimilabillty, 
credibility, completeness, all subjective asaessments made by the staff. 
Although none of these studies provide apeclflc criterion for the evaluation 
of library performance, they do provide some indication of how one may 



begin to isolate user groups and define user needs. 

The effectiveness of the service or work performed by professional 



and non-profeaBional personnel can be measured in terms of goal achleve- 
meht and user satisfaction. The degr^^ and types of services provided by 
the library should be chosen because they achieve a given goal or level 




of user aatlsf action most effectively. A special library naturally 

offer more bibliographic services such as literature searches, preparatiori 
of abstracts and bibliographies and translations than is possible for a 
public or college Tibrary with a much larger and more general clientele. 




Most rocornrnsndstlons for ths suiount snd kind of ssrvicss srs bsssd on 
user needs and wants | problems often arise first in determining wants 
and needs and then publicizing the availability of services when they are 
provided. Pundsack (1955) suggests a library committee of users to 
improve communications and suggest improvements along with talks and 
brochures by librarians. Such a system would at least provide a formal 
channel for determining wants and needs. A typical study is the one done 
by Shank (1959) where members of the academic research staffs at Columbia 
University were surveyed as to their opinions on the location of technical 
reports and the provision of current awareness services. The various 
user requirement studies mentioned have also been used to determine levels 
of service. 

There must also be ways of measuring the effectiveness of the services 
offered in order to give the staff some feedback. Bare (1966) suggests 
a neutral survey team and to ensure wide participation the survey questions 



should be capable of being answered with yes or no responses, request 



demographic data, and include some personal interviews. A computer 



operated system (SURF - Support of User Records and Files) is described 
by Wallace (1966) as a means of identifying current user requirements and 
providing feedback in a fechnical library. In a proposal by Hattery (1963) 
cphtinuous feedback relating to user rsati's suggeated as the 



ultimate cfiterion, for planning.? By extension then the ultimate criterion 
-jf iibrafy performance would degree to which user needs are met . 

Zariversely,- Taube (1959) claims that the effectiveness of a scientific 
inf ormatibn :service cannot , be measured by consumer acceptance ^ince th^ 



service ia profeaslonal rather than consumer oriented. He reviews the 
literature on use atudles, both qualitative and quantitative and concludes 
that none of them provide any prescriptive criteria. 

Employment of the user satisfaction criterion to measure library 
performance creates certain fundamental problems that cannot be completely 
avoided. 

(a) There is always a strong subjective element in such an evaluatlonj 

(b) There can be very few if any comparative atudies done employing 
user satisfaction criteria unless very expensive testing is 
first conducted on the test Instrument, because individuals 
may interpret the scale or values used to indicate degree of 
satisfaction in different vjays, 

(c) There is always a question of defining what is relevant and 
the universality of such a definition, 

(d) There is always a problem in getting a representative sample 

of actual and potential user to respond to any survey queationnaire 

Nevertheless, user satisfaction must be considered one of the primary 
measures of library evaluation. There is a great deal of consensus that 
the ultimate test for any library is user reaction and satisfaction, and 
regardless of any other measures, if the users reject the services and 
fnaterlals offered^ the library must be considered a failure. User 
satisfaGtlon is an adequate criterion of effeGtiveness only when it is 
employed with a full understanding of its limitations and in conjunction 
with other criterion, ‘ 



The use of response time as a measure of library performance is a 
relatively new development- Its use is increasing as many Invest'ig^^ 




view this as one of the more objective msasures and one that is subject 
j to easy quantification. While response time can be measured, there is a 

I great deal of room in which to vary the points at which to make the time 

f; _ . 

f measurement and for that matter whether or not to use real, elapeed, 

[i or some ’’average" time for different situations. As with the other 

I 

I criterion it is possible to use this measure in several different iijays; 

i 

I (a) To measure the time required to secure a copy of a specific 

I document. 



I Cb) 

I 



To meaeure the time required to secure a specific piece of 
information or have a given service performed. 



I In a fairly comprehensive study, Monroe (1D62) indicates that methods 

I " - 

i of evaluation should be based upon a determination of a) library objectives^ 

I - 

1 b) services renderedj and c) the procedures employed in providing the 

I 

I services. In order to evaluate any aervice in terms of the above ^ Monroe 

I suggests using the response time of the operations involved as the best 

I measure. As with so many studies or suggestions for studies this idea 

has some merit but will not do the entire job. One must question whether 
response time is equally important to the performance of each operation 
and to what extent other factors enter into consideration? 



In the case;of jourhals, a shorter response time and increased 
circulation has been recorded in special libraries by Kopkin (1955) and 
Randaii (1954) by first routing the tables of contents, then the requested 
r magazines. The delay in response time caused by the circulation process 

■ .has been discussed by, Elvers (1962) , trie time lag in book returns betting 

i mentioned as one of the problem areas. Circulation service would seem^^C 




to be an area in >which response time would be, of primary interest for 




evaluation purposes. Mathieu (1959) observed with a stop watch some 
operations in a newly established docurnentation center, then compared the 
times for filing, classifying, abstracting, etc. , with other documentation 
centers. In this case, time was a criterion for the performance of a 
manual task rather than overall library performance. In the case of 
current awareness service being measured in terms of response time, we 
see a not infrequent occurence in evaluation of library performance. That 
is, a valid criterion of performance is employed, hut no consideration is 
given to the question of hov; important that particular criterion is to the 
service performed.. The problem is that for the above examples there is 
really a question as to whether performance and service has been Increased 
by decreasing response time. Circulating a list of titles of articles 
rather than journals will no doubt reduce the time factor but as anyone 
with even limited experience knows, the titles of articles are not always 
indicative of the content which would cause more requests for items to 
examine the content. Is that really a step forward In improving current 
awareness? Reaponse time Is Important but if all the criterion of 
performance were applied to that service, it seems likely that in a ■ 
weighted value syatem, response time would receive one of the lower 
weights. 

Another approach considers response time versus utility, services or 
hodlings as a measure of library effectiveness, Anderson (1965) and 
others have used slmulatian techniques aiiowing manipulation of the " 
operating variablea of a system to measure performance. In defining 
elements which can be measufed in a simulation model. Blunt (1966) 




f 

includes the time lapse between the statement of information and reception 
of output as a measurable indication of the system’s effectiveness. This 
is basically what is involved in the Orr (1968 b) study and in the Weasel 
(1969) proposal to use document delivery time and elapse response time as 
a measure of performance. 

Data gathered by Bush (1956) in cases of failure to meet demands for 
materials provides information on the suitability of the loan period, the 
efficiency of the internal processing required for each item, and the 
extent to which the distribution of library materials among branch 
libraries causes a response time delay. The basic assumption is that a 
delay is undesirable and reflects poor performance. There would be reason 
to question such an assumption in some cases, i.e., delay for one class of 
user may result in greater accessibility for another class of user, again 
reflecting the need for a means of applying all the relevant criterion 
employing some weighting system that reflects the criterion's Importance 
to achieving a specific goal. 
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Thompson (1962) argues that the real value of information lies in its 
availability to the user when he needs it. He expresses the value of the 
information as a function of time, e.g. , as a ratio of the time required 
to make the information available to the total time it would be of value 
after a 'need occurs. While this idea has a very basic appeal, one must 
be careful about accepting it too readily. The /primary flaw is that in ; 

reality, the system is concernsd with user satisfaction although everything 




to determine the length of flme 

user would 





i 

} 




not be i^illing to commit himself too strongly on the time factor unless • 
(1) he Is very certain he has to have the information, . in which case 
access and not time is the critical factor; or (2) he is fairly certain 
that the information will not cause him to undertake radical changes. 

In all other cases, which would be true most of the time, the user would 
wish to see the information or document first before deciding on the time 
value. What is likely to occur is that the Information or document that 
turns out to be not quite what was expected will be given a short time 
value (’ If I had known how poor it was I would not have asked for it. I 
wasted a lot of time waiting for nothing."). By the same token, the "good" 
information is worth waiting for In moat peoples’ eyes--if they cannot 
have immediate access to everything. 

while response time is a quantitative criterion of library performance 
and has a high degree of objectivity when certain perameters are agreed 
upon, it tends to be over evaluated as the measure of performance. 

Although there is no evidence to support this suggestion, it seems quite 
reasonable that most users would prefer to have access to more materials 
than to reduce the number accessible and Increase the response time for 
those that are available. In theory, all medical libraries have access 
to all the national medical resources and it is simply a question of how 
long does it ta^ the material. In reality most people will 

admit that this is not quite the case . Until such time that theory and 
preact ice are identical, to suggest that response time is the criterion 
,;of library performance is to be unrealistic. _ There are other factors 
■that must be_ considered. User satisfaction may fall off even if response 




time ia improved if the aystem allows too many error factors to become 
operative. Costs would rise and could skyrocket to the point that the 
service would become uneconomical. Can a system based on response time 
alone be operative for all users, for all needs; it seems likely it could 
not because of the cost and the overloading that could occur in areas of 
high demand. As with the other criteria, response time is best considered 
as part of the total system of evaluation, with a weight assigned to the 
time factor eomensurate with its value in achieving a specific goal or 
service objective. 



2,6 Cost -Benefit 

The final criterion to be dlacussed is a relative newcomer to the 
field of llbrarianshlp. Only a very few studies have been completed that 
have applied the term cost-benefit analysis to their procedures and have 
followed the procedures develaped in buslnesa and government in applying 
this method. There have, in addition, been a number of less sophisticated 
studies of the type that can be considered CDSt-benefit analysis. Since 
these represent older studies, they will be discussed first. The section 
will conclude with a diacussion of some of the more thorough cost-benefit 



studies- 



w 



Purdue University inyestiga^^^ the costs of prbviding library 



, services toy university/ researchers sponsored by outside 'organisations , 
■ Quatmah , (1962) estimated ratio of costs for sponsored research to 






y costs ' for undergraduate library service, then checked; the figures in a 
; C The -aim -vpf? the study was to determine if library , 
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i- . allocations,, to ,use;r classes; were equxtable; , This, is at best a very ; ; \ , ; 
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elementary cost-benefit analysis but it does attempt to examine costs 
in terms of the service provided. A ratio of expenditures to users and 
service has also been used to determine effectiveness. A cost study of 
an industrial library (Taylor 1961) revealed that 32.6% of staff time was 
spent dealing with inquiries and providing information - this accounted 
for 28.1% of the library’s total expenditures. Another approach to 
library budgeting is based on services , by dividing the calculated coats 
of all services into fixed and variable groups, Maybury (1961) claims that 
an accurate picture of the library’s needs will result. Performance 
budgeting was instituted by the Washington, D. C. Public Library as 
reported by Budlngtan (1959) ; he believes that this method of concentrating 
on the character of the work performed by each functional division has 
more merit than simply listing costs under categories of materials and 
services purchased. This method of budgeting requires a thorough knowledge 
of the services to be performed and the amount of use that each service \ 
receives. 

Many researchers have attempted to approach library evaluation 
through combining two or more of the measures. Recognizing that libraries 
have very different specific missions, this approach does not in general 
allow one to compare institutions but rather relates one aspect of 
aervlc’e In a particular library to another service in the same library. 

This approach can be seen most clearly in the studies relating coat to 
benefit. Comparisons between Instltutiona are possible only when each 
library employs identical systems which is seldom the case. However, 
cost^benefit can provide some indication of the relative performance of 



different libraries. Thorne (1955) evaluates a retrieval system in terms 
of cost-benefit, where efficiency in the number of successful searches 
is the benefit and a superior system is one which operates at the Lowest 
cost with the greatest benefit. However, there is no general agreement 
on the definition of a successful search, a problem noted earlier. 

While the method was used in a special retrieval system, it is of course 
applicable to the library situation. In another study investigating the 
value of a retrieval system, Mueller (1959) felt a library should strive 
to achieve a minimum of a 1:1 ratio of man hour costs saved to the cost 
of retrieving documents. He also compared retrieval costs with stsff 
consultation, another way of obtaining needed information. 

Goffman (1964) measures performance in terms of effectiveness 
divided by cost. Formulas are provided in order to show effectiveness 
as a function of cost and time. For an indjlcatlon of past purchaaing 
effectiveness, Hodgson (1959) presents a formula based on cost which 
shows that too much material was borrowed on Inter-library loan when it 
would have been cheaper to purchase the requested items. 

Other studies have compared the costs of two comparable methods of 
operation. The benefit in such cases is a lower cost. An example of 
this approach Is Linder’s (1965) comparison of document indexing and 
book cataloging , Factors contributing to the; greater cost of cataloging 
are noted and the different functions of the too systems described. 

In the same manner, the Library School at; State University (1966) 

compared the operating conditions of three systems of Information 

retrieval, b punched cardv file, a handbook reproduced from that file. 



and conventional cataloging-reference approaches to the same material. 

Cost was one of the criteria and was found to be much higher in terms 
of input and cost per use for a mechanized system. 

A recent book by Raffel and Shishko (1969) is the most complete study 
of cost-benefit in an academic library setting that was encountered in 
the literature search. They start with a program budget for an academic 
library and then move through a series of different coat-benefit analyses 
for each of their basic programs. Clearly, there Is much to he gained 
by using this method. 

Cost-benefit analysis could be considered at least in theory, as 
the criterion of library performance. This would be possible, however, 
only if a large number of different benefits were considered (for 
example, cost-accessibility, cost-use, cost-user satisfaction, cost- 
response time) . Within each such division there would have to be a 
number of sub-analyses in order to cover various situations. While cost- 
benefit seems to have a great deal of promise, it cannot get away from 
the subjective, imprecise, ineonsiatent , difficult to measure factors 
that create problems for most of the other criterion that have been 
discussed. The benefit can be almost anything- -which in Itself provides 
a great deal of room for the variation and slight modifications that make 
comparisons difficult if not impossible to make. For all of the potential 
dif f tcultles , cost-benefit seems to be a very Important criterion for 
measuring library performance. 



3.0 SUMMARY OF LITERATURE SEARCH: MEASURES OF LIBRARY EFFECTIVENESS 



During our literature review vie encountered a variety of measures 
that were employed in attempts to evaluate library performance. Most 
of the measures we considered to be slight modif ications of Oiie of the 
six criterion that we employed to organize our presentation. The complet 
list that follows. Indicates the variant and specific measures that have 
been included under our more general heading. 



I. ACCESSIBILITY 

1. Number of services and degree of services provided various 
classes of users. 

2. Ratio of services requested to services available. 

3. Ratio of holdings to total user population (actual and 
potential) . 



II. COSTS 

1. Staff size. 

2. Staff skill and characteristics. 



3, Unit cost. 

4. Ratio of book budget to users. 



III. 






USE 

1 . 

2 . 

■3. : 

4. 

5. 

6 . ' 



Gross use of services Creference questions answered, 
bibliographies completed, etc.). , 

Ratio of actual users to potential users. 

Total library use. 

Ratio of service use to total number of users. 

Ratio of total use t^^ total number of services. 

Percentage of materials used (by type) and by class of 
users (.student, teacher, researcher, etc;) . 



7 



7. 


Ratio of documents circulated to various classes of users. 


8. 


Ratio of documents circulated to active users. 


9. 


Ratio of total use to total holdings. 


10. 


Item-use-day. 



IV. USER SATISFACTION 



1. 


User satisfaction. 


2. 


User activities (purpose) in library. 


3. 


Percentage of items in collection as listed in some 
checklist. 


4. 


Percentage of items in collection by type of material 
(books, serials, reports, etc.). 


5. 


Percentage of items in collection by type of material 
compared to various classes of users. 


6. 


Quallty-vcilue of items in collection based on expert 
opinion. 


7. 


Ratio of documents used to materials requested. 



V, RESPONSE TI^iE 



1. 


Speed of service. 


2 . 


Ratio of number of services offered to average response 
time for all services. 


■ - 3. 


Ratio of response time (to secure document) to total 
time document is of vaTue. 


4. 


Ratio of holdings to response time. 



VI. COST-BENEFIT 





Ratio of services to cost. 

. ■ , . ■ . . ’ ,, , , , : ' 


'2.:, 


Ratio of total service expenditures to users (actual 




and/pr potential). 


4 . 


Cost-benefit 

■ Cost-response ■ timev^" ^ \r. 




' ■ '''’'■'■■■Jr.''' ■• ■ -:--;■'■■’ ■■-■■"■■■■■ 'J''- . -■. . . ' , ■.; 




^ ~ „■.>; J' ^ '■■■■- -3S- ■" , ■," ;',v- ■ '■■ '■■ ' ’■ 



Several studies have made an effort to include all possible factors 
pertinent to library effectiveness in their evaluations. An example is 
an on-site study of mechanization in DOD libraries (1966) where the list 
of measures covers all those diecuased here. Carnovsky (1955) discusses 
the accumulation of data that has been used to evaluate public libraries 
and warns that they cannot be applied arbitrarily. Each librarian must 
decide on his library’s goals, based on potential readers, their needs 
and desires, and evaluate on the basis of those goals, not on the goals 
of other libraries. An old article by Purdy (1942) summarizes many of 
the basic problems in evaluation of libraries. Wilson and Tauber (1956) 
summarize the past methods used in over-all university library evaluation, 
showing that the principal technique has been comparison. Four approaches 
which have been taken are comparing the present condition of the library 
with that of past years, comparing the library to other aspects of the 
university, comparing the library with other similar libraries, and 
comparing the library with certain standards. They feel that true evaluation 
of a library can only be determined by the extent to which the users 
achieve their purposes, not by comparative techniques. 

3.1 Areas of Research 

The purpose of the literature review was to look at what work had 
been carried out in the area of evaluating library performance i^ order 
to make recbmmendatiohs regarding areas in which further work might be 
productive. The summaries at the end of the discussion of each basic 
criterion reflect the authors’ gerieral thoughts on these criteria. 




The primary result o£ the literature search was that it indicates 
that a great many variations on a few basic approaches to measuring 
library performance have been tried; but of all the studies examined, 
only one attempted to look at the problem of evaluation from the point 
of view of the total library .(Wessel, 1967, 1968, 1969). Most of the 
studies concentrated on one or two measures of one or two services. 

One completes the review still wondering what measures can be employed 
in the hope of achieving valid resulta. Are all the measures , even the 
six "basic” criterion equally important in measuring all services? If 
not, how can they be weighted to reflect this relative importance both 
to the evaluation of a specific service and to the total library program? 

Research that would aid in answering the questions in the above 
paragraph would seem to be of primary importance. Medical libraries, 
like other libraries, perform multiple services and therefore it seems 
unlikely that any single criterion can be considered completely valid as 
the measure of library performance. When it is possible to apply several 
different criteria, the question of weighting each one becomes important. 

In order to determine what the weighting factor should be, it is clear that 
one needs to know how- important each criterion is in achieving a specific 
library function. As was noted above, the literature in general reflects 
an absenee of consideration of a) the total service program, and b) the 
impoftarice of any one criterion in meeting the needs of even one service 
function. Without such corisideretlon it would seem to be Impossible to 
arrive at a valid evaluation of the performance of the library. 



In light of this, it is suggested that one area of needed research 
should be to develop a technique or instrument to aid in determining with 
a reasonable degree of accuracy what a given library’s services are and 
their relative importance in terms of the total library program, A second 
phase of this problem would be to determine which criteria were appropriate 
to measure the performance of the services and the weight that should be 
assigned to each criterion. For example, does it seem valid to give the 
same weight to response time when considering such services as translation 
and information-reference? While response time would be a valid criterion 
in both cases it seems quite likely that most people would rather see a 
slower response time (less weight) and say more accessibility (more weight) 
v7hen evaluating a translation service. However, until there are instruments 
or techniques to aid in determining this, the matter is subject to debate 
and opinion. 

Another area that ought to be Investigated, if total library performance 
is to be considered, is that of conservation. No studies were encountered 
in the literature search that even discussed conservation as an aspect of 
library performance. Some techniques should be developed for evaluating 
the losses that the library incurs In terms of conservation when the 
library attempts to Increaoe its performance in terms of users. 

In general, it would seem from a review of the literature, that perhaps 
the time has come x^hen less effort should be devoted to developing variations 
in the criteria of performance evaluation in terms of one or two services. 
Rather that efforts be directed toward unifying the work that has been 
done in order to evaluate the entire library and perhaps arrive at the 
point at which intelllgeht comparisons between libraries may be made. 




4 0 UNIVERSITY CITY SCIENCE CENTER STUDY 



In addition to reviewing the literature relating to the measurernent of 
library effectiveness , and preparing a summary list of measures that have 
been employed as Indicators of library performance, the investigators 
also carried out some additional statistical analysis based upon data avail- 
able from the National Survey of Medical School Libraries (PH 43-60-94) . 

That survey was prepared for the National Library of Medicine by the 
University City Science Center (UCSG) of Philadelphia. In the course of 
their research a great deal of statistical data was gathered. As Is always 



the case, data lends itself to many more types of analysis than need be 
done in any one study. We proposed to re-examine the data from some different 
points of view, and through the cooperation of NLM and UCSC the computer 



tape file of the data was made available.* 

The UCSC study Is in many ways typical of all studies of library eval- 
uation. It focuses on only one or two aspects of library service, it applies 
a single criterion of evaluation to a service with no attempt to justify the 
significance or validity of the criterion measured for that service, and the 
study does not consider the relation of the service studied to the overall 



purpose and function of the library, in this case medical school libraries. 
These are, however, as indicated in the report on the literaiure review 
rather common fallings, and perhaps a failing only when considered from a 
much broader perspective than most studies assume. However, it would seem 



* The investigators wish tp express their appreGiation to UCSC for their 

cbpperation which , indeed eohslsted of more' than simply supplying a tape . 
They patiently ahswered cjuestions and helped us make efficient use of their 
material. This kind of sciehtlfic sharing is still far from common, and 



that any study that does not at least note the relationship between the 
problem under investigation and the total system of which the problem is 
part, would have a rather limited value. 

The objective of the UCSC study was ”to determine, analyze and report 
the service capability and performance of all active medical school (training 
toward the M.D.) libraries in the United States.”* In order to accomplish 
this objective, the study undertook "to administer a Document Delivery 
Test (DDT) appropriate to academic medical libraries to 92 medical school 
libraries and 3 reservoirs.”** (reservoir libraries). It is Important to 
note that the DDT was not ready for operational use, but was still an 
experimental technique that required further testing. 

4.i The Valid i tv of the DDT 

The DDT essentially consists of a list of 305 document (304 for the 
reservoir libraries) references. Effectiveness in this test is determined 
in terms of the speed at which the library can deliver the documents to a 
requester. The references are taken from the field~of medical research and 
not medical education. The two subjects are related, but they are not 
identical and there is no reaspn to assume a priori that the performance of 
a medicBl school library can be fully and adequately measured by how fast 
it can deliver a few hundred research reports. Surely a medical school 
library serves a broader spectrum of needs than just research. No 

* University City Science Center. Final Report - National Survey of Medical 
School Libraries . University City Science Center, Philadelphia, 1968. 

*'* Ibid. P. 2. 




justification or even suggestions are made regarding the validity of the 
sample in either the final report or the published articles, with only one 
small exception. The exception states, "because researchers’ needs for 

primary source documents pose one of the most severe demands on a library’s 

/ 

capability, we do not consider this limitation a critical weakness for a 
test of the document delivery capability of academic biomedical libraries."* 
As a test of research document delivery capability the DDT may be valid;' 
however, in terms of meeting the study’s objective "to determine, analyze 
and report the service capability and performance" the test does not appear 
to have even surface validity. In order to validate the DDT in terms of 
research,, documents would require drawing tnd testing a new sample at the 
same libraries. This has not been done. 

In addition to the validity of the test itself, the validity of using 
response time, or document delivery time, as a measure of the total service 
and performance capability of medical school libraries is certainly 
questionable. Surely there la more to the performance of medical school 
libraries than the speed at which documents are supplied to users. Granting 
the premise that speed is an Important factor, as indicated in the literature 
survey section, this factor can be used to study the performance of a wide 
range of services, not just the document delivery. In addition the UCSC 
Investigators make no attempt to justify the DDT as the sole measure of 

* Orr^ Mchard H. , Pings, Vern M. Irwin H. , Olson, Edwin E, , and 

Carol C. "Development of Methodplogio Tools for Planning and 
Managing Eibrary Seryicesj ll . Measuring a Hibriry ’ s Capability of 
Providing Docutnents," Bulletin of the Medical Library Association 56, 

' (July ^1968) ■■:p:.T257 V/;:::::.- :r-.„ V-;: V. 



peirf orrnance and service capability. In Einy case the entire question of the 
validity of concept needs to be re-evaluated. 



4.2 Data Analysis DDT Reliabilityi 

Aside from the validity, the reiiablllty of the DDT had not been reported 
However, a split -half reliability could be determined by reanalyals of the 
raw data, and this was one of the tasks undertaken by the present study. 

The sample of 305 documents employed in the test were split into two groups. 
The method of splitting was based upon the number assigned to the term, all 
even numbered items in one group (XS2 documents) and the odd number in the 



other (153 documents) . The performance index (PI) for each institution 
was recomputed for each option for each of the samples. We employed the 



UCSC method of computing the PI as reported 
(PI = ^ * ) • (PI ~ performance Index, 5 “ 



in the UCSC progress reports 
the number of possible delivery 



speeds, m.s. = the mean speed for all documents at an institution, and 
4 — a constant employed to insure a decimal PI. One problem encountered 
was a lack of specific information in the final UCSC report about the pro- 

- ■ n- 

cedures used, so that we could be certain we were working from the same 
basQ.) The Inatltutlons were then ranked according to their PI on option 1 
and 4 for the odd and even samples. These rankings were compared to one 
another and to the PI as computed by UCSC based on the total aample. 

coefficient analysis are as fpllows. 







CORRELATION COEFFICIENTS FOR DDT RELIABILITY SPLIT HALF 

I 

I 

I 

I 



/ 

/ 






Options 








/ 

/ 

/ 


1 


2 


3 L|. 


5 


6 


mean 


1 

Regular (odd -even) 


.93 


93 


.94 .94 


.92 


.94 


.93 


Reservoir (odd-even) 


.96 


97 


.95 .95 






,96 


RANK CORRELATION COEFFICIENTS 


FOR 


DDT RELIABILITY 


SPLIT 


HALF 








Options 










1 


2 


3 4 


5 


6 


mean 


Regular (odd -even) 


.91 


92 


.90 .92 


.89 


.90 


.91 


Reservoir (odd-even) 


.90 


90 


.93 .94 






.92 



As can be seen from the figures the degree of association is extremeiy 
high. In the case of rank correlation coefficient, we simply ranked the 
institution in terms of its PI for each option using only the even 
numbered documents and then only the odd numbered documents- We then 
checked to determine how closely the two halves came to ranking the 
institution in the same order. Again from a statistical point of view it 
would seem to make little difference which half were used if they would 
produce very similar results. These results may he Interpreted as indicatin 
that the original sample is reasonably consistent and uniform, containing 
no measurable bias. . 



Data Analysis ■» DDT and Categories 
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A second analysis was undertaken to determine whether or not twenty- 
three status categories were necessary in order to record scoring information 
for the DDT. The UCSC study employed 23 different categories for recording' 
the status of a given document (see below for a list of these categories) , 

It seemed probable that a reduction In the number of categories would 

(1) produce results of equal or greater relevance for scoring the DDT, 
and (2) produce a format which would be easier to use in the field, reduce 
possible variations in scoring due to different interpretations of a 
situation, and make scoring easier. Six categories were used in this 
analysis. They were: 

(1) On shelf* - On shelf, to be shelved and on shelf-second search. 

(2) Not in collection - Not in collection. Interlibrary loan, missing, 
can’t locate. 

(3) In storage or special location - Off premises, in storage, special 
location-mediated, special location-not mediated, other known statue, 
other - second search. 

(4-) In clrculatipn - Reserve loan, faculty loan with recall, faculty 
loan without recall, student loan with recall, student loan without 
recall, other loan with recall, other loan without recall. 

(5) In Bindery - In binding. 

(6) In process - In process not available, in process available. 

The categories were given speed codes that differed slightly from those 
assigned in the UCSC study. This was necessary because the UCSC study occu- 
sionally used different apeed codes for categories that we placed under one 



* Covering the following UCSC categories. 
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heading. However, the scoring concepts were kept consistent v;ith the UCSC 
study. For the regular libraries we used the follo^A;ing speed codes for 
option 1. 



Our Code 




UCSC Code 






On shelf 


1 


1 = 2,3 






Not in collection 


ILT® 


ILT, 5, ILT, 


ILT 




In storage or special location 


2 


EDT* **, 2,1,2, 
□pinion 


EDT, 


editor’s 


In circulation 


4 


LP*** 
EDT, 4 ” 


4 ^ 
’ 2 


■ 4t£ 

’ ■ 2 


In bindery 


2 


EDT 






In process 


EDT 


2, EDT 







Using the new apeed codes the PI for each library was rGoalculated 
and compared to the UCSC PI for each section, with the following correlation 
coefficient results. 



CORRELATION COEFFICIENTS FOR UCSC PI AND "NEW" PI 



Options 

1 2 3 4 5 6 mean 

Regular libraries .99 .99 .97 .97 ."4 .91 .96 

Reservoir libraries 1.00 1.00 1.00 1.00 1,00 

These results may be interpreted as indicating that using the simplified 
method of scoring one would achieve practically the same results as by using 
the more complex method. 

* Inter library loan time 

** ■ Estimated delivery time 

*** Loan period 
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Several hypothesea were checked against the data. Each of the 



following hypotheses was checked against the data by employing a rank 
correlation analysis. 

1) There i.s a very high positive correlation between the number of 
documents on the shelf and the overall performance index. 

2) There is a very high negative correlation between the number of 
items not in the collection and the overall performanee index, 

3) There is a high negative correlation between the number of docu- 
ments in storage or special locations and the overall performance 
index . 

4) There is a moderate negative correlation between the number of 
docuii.ents in use and the overall performance index, 

5) There is a moderate positive correlation bett^een the number of 
documents in process and the overall performance index. 

6) There is a moderate negative correlation between the number of 
documents in the bindery and the overall performance index. 

We decided to use only Option 1 of UCSC for testing the hypotheses 
(see section 4,4 for detailed discussion of scoring options). The rank 
correlation coefficient was calculated for the Option 1 PI as determined 
by the UCSC method and each of six hypotheses on the basis of the number of 
documents in each category. The data indicates for regular libraries that 
hypothesis one was correct (+.902). We suggested that there would be a high 
positive correlation between the number of documents on the shelf and the PI. 

In hypothesis two the high negative relationship between the PI and the 
number of documents not in the collection was found to hold (-.858). Again 
we employed the method of PI to the number of documents in the appropriate 
category, in this case not in the collection. The hypothesiEed relationship 
(high negative), suggested in hypothesis three, between the PI and the number 
of documents in special locations did not exist (-.104). A moderate negative 
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relationship was predicted to exist between the PI and the number of docu- 
ments in use in hypothesis four. This was not the case; the correlation 
coefficient value t\?as .067. In the case of hypothesis five a nioderate 
positive correlation was anticipated between the PI and the number of docu- 
ments in process. Again the hypothesis was not supported by the analysis 
(-.007), Finally the relationship posited in hypothesis six (moderate 
negative) between the PI and the number of documents in the bindery did not 
exist (.006). The evidence for the regular libraries clearly indicates 
the only significant relationship is between "on the shelf" and "not in 
the collection" and the PI. This would seem to indicate the DDT and PI 
are relatively Insensitive to the other status categories. 



HYPOTHESIS 


PREDICTED 

CORRELATION 


REGULAR 

LIBRARIES 


RESERVOIR 

LIBRARIES 


1 


+ + 


.902 


.635 


2 


-- 


-.S5S 


-.982 


3 


--- 


-.104 


-.035 


4 


Hh 


.067 


-.235 


5 


+ 


-..007 


.117 


6 




„Q06 


-.225 



\ 



++ = high positive, 

+ = moderate positive. 
— “ high negative. 

- - moderate negative. 
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The same basic pattern was encountered in the reservoir libraries 
although the lack of correlation is not as high. The most surprising feature 
of this analysis was the lack of a high positive relationship for hypothesis 
one. Hypothesis one was rejected because the relationship at best can only 
be considered moderate (,635); hypothesis two was supported (-,1,52); 
hypothesis three was rejected (-.035); hypothesis four was rejected (-,2o5), 
hypothesis five was rejected (.117); and hypothesis six was rejected (-.225). 

In part these results may be a reflection of the fact that documents 
on the shelf or not in the collection represented almost 92% of the total 
sample for the regular libraries and 89.2% for the reservoir libraries. 

As a result the influence of the other status category documents on the PI 
was at best very slight. 

What this means is that in essence the DDT measures how many documents 
are or are not in tlie collection and very little else. Since the correlation 
of books on the shelf and the PI for regular libraries is r - .9, r — .81, 
measuring the 81% of the total variance of the PI score is accounted for 
by this one variable. In view of the questions that exist about the 
suitability and validity of the DDT documents for testing medical school 
libraries, we feel that this Instrument Is of limited use. That is, any 
standard medical bibliography could be used to check the holdings of any 
medical school library and the percentage of documents held by that library 
would provide as much information about the '’service capability and 
performance” of that library as the DDT results. 



4.4 Data i^nalvsis - DDT and Scoring Options 



Throughout the UCSC study there is an emphasis upon scoring option 
four as the significant option to be considered. "Option 4 - score with 
both short-term and long-term activity controlled, representing what the 
performance would have been if all test documents in use within the library 
and on loan were instead ’on shelf "In looking up a library's score. 
Option 4 is the one most indicative of the institution's capabilities, as 
it represents performance with short and long term activity controlled . . . 
Option 4 is the one used in all subsequent graphs and in comparisons with 
other variables."** No evidence is given to support these statements, and 
since Option 1 represents the "raw score"*** for the test, and is thus 
easiest to calculate, it seemed desirable to check on the significance 
or variation, if any, that exists between the various options. 

The null hypothesis (that no significant differences exist between 

any of the scoring options) was checked by means of correlation coefficients 

[ 

for the PI and by rank correlation cojaf f iclents . The results of these 
analyses are* 

CORREMTION COEFFICIENT FOR SCORING OPTIONS - REGULAR LIBRARIES 
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Ibid. , p. 23 
Ibid., p. 24 
Ibid., p. 23 
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KANIC CORRELATION COEFFICIENT FOR SCORING OPTIONS - REGULAR LIBRARIES 
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CORRELATION COEFFICIENTS FOR SCORING OPTIONS - RESERVOIR LIBRARIES 
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RANK CORRELATION COEFFICIENTS FOR SCORING OPTIONS - RESERVOIR LIBRARIES 
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Clearly these date indicate that no significant differences exist and that 
all the options, or scoring methods, are highly correlated with one another 
Therefore the suggestion that Option 4 is ''most indicative of the 



institution’s capabilities" in terms of performance is not substantiated. 

It is no more and no less indicative of the service capability than any 
of the other options. The reason for this result was suggested above; the 
DDT is relatively Insensitive to any status category except "on shelf" and 
"not in collection". In view of this finding it would seem to be unnecessary 
to go to the trouble of calculating nything but the raw score (Option 1) . 

The Orr, et al (1968 a, b, c) articles provide some reasons for using all 
of the options; however, these reasons appear to be based upon the assumption 
that the PI would be much more sensitive to variations in the twenty-three 
status categories. On the basis of the data collected from the 92 "regular" 
and 15 "reservoir" medical school libraries, this assumption did not hold. 



M-. 5 Summary - DDT 

In summary, the PI and DDT, as applied in the UCSC study, does not yet 
appear to be a totally satisfactory method of assessing medical library 
service capability. The reasons for this statement are: 

1) There is no evidence to support the idea that response time is 
the best measure of service capability much less the sole measure, 

2) The sample of documents employed in the DDT does not 'adequately 
reflect the information needs of the total user population served 
by a medical school library (only the "author-researcher" needs 
are represented) . 

3) The scoring methpd (PI) is not sensitive enough (at least for the 
reported libraries) to the various status categories set up by 
the UCSC study./ This makes the test cumbersome to use and in 
fact the efforb is wasted because the results will vary only 
slightly for having done the extra work. 

' ■ i - ‘ 

4) The DDT, because of this lack of sensitivity, is not any more useful 
for determinirig the service capability than the results of checking 
a standard medical bibliography against the holdings of a library. 
The latter operation could also be completed more quickly and 
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cheaply. 
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This is not to say that the method is iv-ithout value. There are a number of 
areas within which refinements could make it a very valuable tool of evalu- 
ation. Some areas where additional work could produce a more valuable 
technique are: 

1) A sample drawn in a manner that would be more reflective of all 
patrons’ needs is essential to valid employment of the DDT as a 
measure of a library’s ability to provide a wide range of users 
with the documents they need. 

2) A reduction in the number of status categories would make the test 
easier to use and would produce results as accurately as the present 
method, 

3) An increase in the values assigned to documents in status categories 
other than ”on shelf” and ’’not in collection” should make the PI 
more sensitive to variations in those other categories. 

4) A reduction in the number of scoring options to one (the raw score) 
i^ould also seem to be appropriate and would reduce the time 
required to calculate the results. Although if there were an 
increase in the sensitivity of the status categorleSj additional 
scoring options might be of some value. 

4.6 Data Analysis - Inventory of Library Services 

All the reviewers of the UCSC final report commented on the surprising 
lack of correlation between the PI and ILS score,. It would seem reasonable 
that a moderate positive correlation should exist between the two factors 
since both measures took into account circulation and inter-library loan 
activity. The proposal we submitted to the National Library of Medicine 
indicated we would attempt several analyses of these data in an effort to 
determine the factors that created the lack of correlation. 

Some time was devoted to attempting to determine how the ILS score was 
calculated by UCSC staff. Since neither the final report, the published 
articles (Orr, et al 1968 c) , nor the progress reports submitted to NLM by 
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UCSC gave adequate information about the ILS scoring techniques and our own 
efforts did not produce satisfactory results, we contacted the UCSC staff. 
After some time, we received written confirmation of the following points 
(see Appendix A) . 

1) The ILS score for an institution is not a composite score based 
upon the responses given to all questions for all five user 
categories. Rather, the score is the highest score achieved in 
one of the user categories. Thus if the house staff at an 
institution received the greatest service score, that score 
represented the ILS ''score" for that institution. 

2) The rationale for the above approach is that since the library is 
capable of giving that level of service to one category of user, 
it should be given that score as the measure of its service 
capability. 

3) The scores for two institutions (003 and 026) represent scores 
achieved on the basis of services rendered, a sixth category of 
user (State Health Officers) that is not mentioned in the final 
report nor its appendices. 

M) The appendix listing the data from the ILS test contains a number 
of incorrect listings making it impossible to duplicate the same 
results as reported in the UCSC study, if the appendix data are 
used. The original data were not available for use. 

5) The published discussion of the I'LS and scoring of that test has 
no relationship to the manner in which this aspect of the study 
was handled in the final report. 

Because of these findings no further attempts to work with the ILS 
material were undertaken. In view of the unusual manner in ttfhich these data 
were handled, it is not too surprising to find they did not correlate with 
the DDT results. The rather questionahle rationale about the highest score 

for one class of user representing the total service capability of a library 

/ 

cannot be accepted/. The assumption that services given to a limited number 

/■ 

ii 

of patrons can beigiven to all patrons is simply unrealistic. Perhaps the 

reworked analysij of the ILS mentioned in Appendix A will remedy many of these 

// 

problems. !• j 




I 



4.7 Summary - 13CSC Final Report 



In summary, the UCSC study, while having a number of questionable factors 
included in its methodology, appears to have started toward developing one 
valid technique for measuring one aspect of the problem of library service 
narformance. The DDT is a useful experimental device, but more work needs 
to be done before it can be put to operational use. More Importantly, 
additional instruments need to be developed to measure the full range of 
library services. 




APPENDIX A 



January 20, 1970 



Mr. G. Edward Evans 
School of Library Science 
University of California 
Los Angeles, California 

Dear Mr. Evans: 

This is to confirm our telephone conversation regarding the scoring 
and reporting of the ILS data. I apologise for the lateness of this 
letter, but I hope that it will be of use to you anyv;ay. 

Your understanding of the procedures as stated in your letter of 
November 7, 1969, tc Mr. Bagley is correct. The ILS score for each 
institution is not a composite score, but is the score of the user 
category receiving the highest degree of service. Thus, the score 
represents the maximum service capability of the institution at the 
time of testing. 

As might be expected, this user category was most often the faculty, 
with but a fev-7 exceptions. In two of these, 002 and 026, the highest 
degree of service was accorded a category of user for which there 
is not a summary sheet breakdown in the appendix of the report, as 
so few institutions mentioned it at all. These two scores were, I 
believe, given to you by telephone. 

I checked with Dr. Orr concerning the published discussion of the 
ILS, and he confirmed that it was concerned with the projected method 
of ^^?eighting and scoring answers and answer categories. This method 
was not developed at the time of our study and is thus not the one 
used in our reporr to NLM. However, I understand that this weighting 
and scoring has subsequently been accomplished and that the results 
thereof are to be published shortly. This method should provide an 
excellent "in depth" manner of evaluating each institution’s service 
capability. 

Best wishes on the progress of your report; I hope you have not 
had further difficulties v\?ith it since I talked with you last. 

Sincerely, 
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(Mrs.) Dana Close 
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